Skip to content

fix(ops): make the alert steps actually run - #143

Open
g5n-dev wants to merge 2 commits into
mainfrom
fix/alert-steps-never-run
Open

fix(ops): make the alert steps actually run#143
g5n-dev wants to merge 2 commits into
mainfrom
fix/alert-steps-never-run

Conversation

@g5n-dev

@g5n-dev g5n-dev commented Aug 3, 2026

Copy link
Copy Markdown
Owner

告警通道从未真正工作过

监控运行 30798868809 正确检测到了故障,然后没有通知任何人

3 failure  Measure publication heartbeat          ← 正确报出 101 小时无新文章
4 skipped  Deliver publication heartbeat alert    ← 告警被跳过
5 skipped  Enforce divergence and stale thresholds
6 skipped  Deliver release state alert

Issue 一个都没创建。检查是对的,通知从来不存在。

根因是我写的 bug

GitHub Actions 的默认 shell 是 bash --noprofile --norc -eo pipefail——已经带 -e

我写的 set -uo pipefail 只是追加 -u没有移除 -e。于是:

set -uo pipefail
python3 scripts/publish_heartbeat.py ...   # 返回 1(critical)
code=$?                                     # ← -e 已经终止,这行没执行
echo "exit_code=$code" >> "$GITHUB_OUTPUT"  # ← 也没执行

投递步骤的条件是 steps.heartbeat.outputs.exit_code != '',output 从未写入,条件为假,skipped

这正是这套监控要消灭的失败模式,被我在上一层原样重现了一遍。

第二个缺陷

同一次运行显示:心跳失败后,发布状态检查也被跳过。一个检查坏掉会掩盖其余所有问题。

改动

  • 每个测量步骤显式 set +e,让退出码能被记录
  • 三个测量步骤加 if: always(),检查之间互不阻断

为什么要测

这个 bug 的症状是沉默——而沉默与健康无法区分,靠看是发现不了的。所以加了四条测试固定这两个属性:

  1. 任何写 exit_code 的步骤必须有 set +e
  2. 任何 Deliver 步骤必须 always()
  3. 三个检查互不阻断
  4. job 自身颜色仍如实反映四项信号(坏掉的告警通道不能把故障粉饰成健康)

全量 1,377 项通过

说明

这是今天第四次「我自己造出要修的病」:容量报告曾把 CRITICAL_RATIO 硬编码成 retention 主动维持的 0.85;这次是让告警在检测成功后静默。共同点是——只验证了检查逻辑,没验证信号能否送达

🤖 Generated with Claude Code

t and others added 2 commits August 3, 2026 19:46
Run 30798868809 detected a real outage and told nobody. The publication
heartbeat correctly exited 1 after 101 hours without a new post, and the
delivery step was skipped.

GitHub's default shell is `bash --noprofile --norc -eo pipefail`, which already
sets -e. The steps declared `set -uo pipefail`, which adds -u but does not
remove -e, so the non-zero exit aborted the step before `code=$?` and the
`echo "exit_code=..." >> $GITHUB_OUTPUT` ever ran. The delivery step guards on
`steps.<id>.outputs.exit_code != ''`, which was therefore false, so it skipped.

The check worked. The notification never existed. That is precisely the failure
mode this monitoring was built to eliminate, reintroduced one layer up.

The same run shows a second defect: the release-state check was skipped too,
because a failed heartbeat aborted the remaining steps. One broken check hid
whatever else was wrong.

Disable errexit around each measurement so the code is recorded, and let the
checks run independently of each other.

Four tests pin both properties, since the symptom of getting this wrong is
silence — which is indistinguishable from health, and cannot be noticed by
watching.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant